Practical Prediction of Inductive Generalization by the Method of Random Falsifiability

The relationship between learner capacity and generalization, established by Vapnik’s uniform convergence framework, suggests a remarkably simple validation procedure: reject the inductive hypothesis if the learning algorithm can also store the training examples with random labels. We study the properties of this apparently coarse estimator, which can be considered as the simplest possible capacity control method. First, the method is theoretically derived, under reasonable assumptions, from the uniform convergence requisites for generalization. Then, exhaustive experiments are performed on synthetic and natural data. Finally, we characterize the kind of learning situations in which the validation method is appropriate. We also discuss some implications with respect to the theoretical limits of pure induction. In summary, we will show that the method of random falsifiability is a satisfactory qualitative generalization predictor for learning machines with controllable capacity that try to minimize the empirical error. It is especially useful in adverse situations in which the problem distribution and the properties of the learning machine (e.g. its VC dimension) are unknown and additional test examples are not available.